<html>
<head><meta charset="utf-8"><title>binary format versioning · t-compiler/wg-self-profile · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/index.html">t-compiler/wg-self-profile</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html">binary format versioning</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="164238095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164238095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164238095">(Apr 26 2019 at 07:02)</a>:</h4>
<p>We might want to look into supporting different versions of the binary format. The specific encoding might change but it would probably not be too hard to make measure support different encodings at the same time. Then people would not need to be so careful about using tools matching the compiler.</p>



<a name="164238109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164238109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164238109">(Apr 26 2019 at 07:02)</a>:</h4>
<p>At a minimum this would allow us to give a sensible error message if a tool can't handle a file.</p>



<a name="164238142"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164238142" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164238142">(Apr 26 2019 at 07:03)</a>:</h4>
<p>Besides, it would seem that it's more important that the compiled summarize binary uses the same version of measureme, as the one in the compiler being measured, than that summarize is compiled <em>by</em> that compiler?</p>



<a name="164238408"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164238408" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164238408">(Apr 26 2019 at 07:09)</a>:</h4>
<p>Perhaps one should keep master up to date with the measureme in nightly/rust master, so this matches up every time you pull both from master?</p>



<a name="164238483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164238483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164238483">(Apr 26 2019 at 07:10)</a>:</h4>
<p>(or just don't mess up support of previous formats or something of that kind)</p>



<a name="164239121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239121">(Apr 26 2019 at 07:25)</a>:</h4>
<p>I imagined that measureme would keep supporting old formats</p>



<a name="164239209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239209">(Apr 26 2019 at 07:27)</a>:</h4>
<p>and yes, what you say is true: the measureme versions have to match up. what compiler the tools are compiled with doesn't matter</p>



<a name="164239278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239278">(Apr 26 2019 at 07:28)</a>:</h4>
<p>But at least if we support older formats the measureme version would just have to be larger or equal in <code>summarize</code></p>



<a name="164239298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239298">(Apr 26 2019 at 07:29)</a>:</h4>
<p>yes</p>



<a name="164239308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239308">(Apr 26 2019 at 07:29)</a>:</h4>
<p>but if we stick to a stable encoding of just the version identifier then older versions can at least tell that they are dealing with a newer encoding</p>



<a name="164239367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239367">(Apr 26 2019 at 07:30)</a>:</h4>
<p>That would also be really nice</p>



<a name="164239405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239405">(Apr 26 2019 at 07:31)</a>:</h4>
<p>I'd probably just add a small header to each of the files</p>



<a name="164239417"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239417" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239417">(Apr 26 2019 at 07:31)</a>:</h4>
<p>some magic 4 byte tag and 4 little-endian bytes for a version number</p>



<a name="164239473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239473">(Apr 26 2019 at 07:32)</a>:</h4>
<p>Yeah, 32 bits for the version sounds enough</p>



<a name="164239495"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239495" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239495">(Apr 26 2019 at 07:33)</a>:</h4>
<p>definitely :)</p>



<a name="164239670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239670">(Apr 26 2019 at 07:36)</a>:</h4>
<p>Do the files have any tag or similar now?</p>



<a name="164239806"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164239806" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164239806">(Apr 26 2019 at 07:39)</a>:</h4>
<p>Also, I was wondering: why is a profiling three files instead of bundling it in one? It seems like having one file would improve usability for the user.</p>



<a name="164240009"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240009" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240009">(Apr 26 2019 at 07:43)</a>:</h4>
<p>the files don't have a tag or any kind of header at the moment</p>



<a name="164240024"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240024" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240024">(Apr 26 2019 at 07:43)</a>:</h4>
<p>using three files allows us to be a bit more efficient when generating the data</p>



<a name="164240088"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240088" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240088">(Apr 26 2019 at 07:44)</a>:</h4>
<p>I think at least :) maybe the circumstances have changed since that decision was made, we should validate that</p>



<a name="164240121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240121">(Apr 26 2019 at 07:45)</a>:</h4>
<p>however, ideally it shouldn't matter if there's one or three files because the tooling should hide this implementation detail</p>



<a name="164240213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240213">(Apr 26 2019 at 07:46)</a>:</h4>
<p>I mean my point is you can't really hide that detail. There user has three files in their directory.</p>



<a name="164240619"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240619" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240619">(Apr 26 2019 at 07:55)</a>:</h4>
<p>yeah, I imagine that that can be inconvenient when having to get rid of those files manually</p>



<a name="164240629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240629">(Apr 26 2019 at 07:55)</a>:</h4>
<p>but those binary files are not really meant to live long or be copied somewhere</p>



<a name="164240695"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240695" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240695">(Apr 26 2019 at 07:56)</a>:</h4>
<p>they are a means of serializing the profiling data while the compiler runs, in a way that has as little overhead as possible, which is important in order not to skew the measurements too much</p>



<a name="164240717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240717">(Apr 26 2019 at 07:57)</a>:</h4>
<p>once all profiling is done and the system has resources for the postprocessing step, the binary files should be made obsolete asap</p>



<a name="164240777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240777">(Apr 26 2019 at 07:58)</a>:</h4>
<p>all of this should be handled by a tool so that the user doesn't have to deal with it</p>



<a name="164240798"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164240798" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164240798">(Apr 26 2019 at 07:59)</a>:</h4>
<p>what exactly this tool looks like we don't know yet :)</p>



<a name="164241322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241322">(Apr 26 2019 at 08:08)</a>:</h4>
<p>I mean, in principle you could just convert them immediately after the compiling finishes, at which point they no longer skew the profiling. On the other hand, doing IO at all during profiling sounds like it would skew it unnecessarily compared to just keeping it in memory? Maybe. I don't know how large the data is.</p>



<a name="164241567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241567">(Apr 26 2019 at 08:13)</a>:</h4>
<p>I think for the regex crate (which is medium-sized) it was something like 20 megabytes</p>



<a name="164241631"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241631" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241631">(Apr 26 2019 at 08:14)</a>:</h4>
<p>it would certainly make sense to write this data to a RAM-disk of some kind</p>



<a name="164241692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241692">(Apr 26 2019 at 08:15)</a>:</h4>
<p>You could even pre-allocate a large buffer like 100 MB and write it there, then flush to a file when we're done. Then if the buffer is filled, allocate another 100 MB buffer in some sort of rope to avoid the cost of copying during profiling.</p>



<a name="164241757"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241757" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241757">(Apr 26 2019 at 08:16)</a>:</h4>
<p>or if you really need to flush it to a file, try to profile how much time that took?</p>



<a name="164241776"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241776" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241776">(Apr 26 2019 at 08:17)</a>:</h4>
<p>we are using a memory map on Unix at the moment, which is a lazily allocated 1GiB buffer</p>



<a name="164241791"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241791" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241791">(Apr 26 2019 at 08:17)</a>:</h4>
<p>on Windows it turned out that writing to file immediately performs the best</p>



<a name="164241859"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164241859" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164241859">(Apr 26 2019 at 08:18)</a>:</h4>
<p>the memory map allows us to not do any locking during event recording</p>



<a name="164245200"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245200" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245200">(Apr 26 2019 at 09:10)</a>:</h4>
<blockquote>
<p>I think for the regex crate (which is medium-sized) it was something like 20 megabytes</p>
</blockquote>
<p>Regex is actually 75mb (at least on macOS but I'd imagine all of the platforms are similar)</p>



<a name="164245238"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245238" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245238">(Apr 26 2019 at 09:10)</a>:</h4>
<p>It zips down to 12mb though</p>



<a name="164245251"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245251" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245251">(Apr 26 2019 at 09:11)</a>:</h4>
<p>Uhh, on my linux laptop it is only 3.4 MB</p>



<a name="164245261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245261">(Apr 26 2019 at 09:11)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> was that before event filtering?</p>



<a name="164245263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245263">(Apr 26 2019 at 09:11)</a>:</h4>
<p>Huh</p>



<a name="164245266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245266">(Apr 26 2019 at 09:11)</a>:</h4>
<p>Yeah</p>



<a name="164245276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245276">(Apr 26 2019 at 09:11)</a>:</h4>
<p>Is the default all events?</p>



<a name="164245278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245278">(Apr 26 2019 at 09:11)</a>:</h4>
<p>My number is just the size of the three files created by <code>cargo +nightly rustc -- -Z self-profile</code></p>



<a name="164245323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245323">(Apr 26 2019 at 09:12)</a>:</h4>
<p>Or all - incremental hit?</p>



<a name="164245366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245366">(Apr 26 2019 at 09:12)</a>:</h4>
<p>the default is everything except for query cache hits, I think</p>



<a name="164245374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245374">(Apr 26 2019 at 09:12)</a>:</h4>
<p>Oh ok</p>



<a name="164245376"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245376" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245376">(Apr 26 2019 at 09:12)</a>:</h4>
<p>Ignore me then</p>



<a name="164245387"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245387" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245387">(Apr 26 2019 at 09:13)</a>:</h4>
<p>How did you get 75 MB though?</p>



<a name="164245400"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245400" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245400">(Apr 26 2019 at 09:13)</a>:</h4>
<p>That event is like 85% of the file</p>



<a name="164245425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245425">(Apr 26 2019 at 09:13)</a>:</h4>
<p>When all events are turned on, it's something ike 3.5 million events out of ~4 million total events</p>



<a name="164245435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alice Ryhl <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245435">(Apr 26 2019 at 09:13)</a>:</h4>
<p>How do you turn on all events though?</p>



<a name="164245510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245510">(Apr 26 2019 at 09:14)</a>:</h4>
<p><code>-Z self-profile -Z self-profile-events all</code></p>



<a name="164245625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/164245625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#164245625">(Apr 26 2019 at 09:16)</a>:</h4>
<p>(This is also a clean compilation which is the worst case)</p>



<a name="165242977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165242977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165242977">(May 09 2019 at 11:14)</a>:</h4>
<p>I'm going to look into implementing this...</p>



<a name="165243004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165243004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165243004">(May 09 2019 at 11:15)</a>:</h4>
<p><a href="https://github.com/rust-lang/measureme/issues/40" target="_blank" title="https://github.com/rust-lang/measureme/issues/40">https://github.com/rust-lang/measureme/issues/40</a></p>



<a name="165250419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165250419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165250419">(May 09 2019 at 13:08)</a>:</h4>
<p>Hm, it looks like keeping to support older versions of the binary format is a bit more complicated than I want to implement right now.</p>



<a name="165250799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165250799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165250799">(May 09 2019 at 13:13)</a>:</h4>
<p>The <code>StringRef</code> is tied to the binary encoding of the string data. In order to solve that there are a few options:</p>
<ul>
<li>Make <code>StringRef</code> more eager, potentially making things less efficient</li>
<li>Turn <code>StringTable</code> into a trait with an associated <code>StringRef</code> type. That would mean that all consumers of events would have to be made generic. Yuck.</li>
<li>Convert old data to the newest format when encountering, which makes loading old data less efficient. Also, it seems error prone.</li>
</ul>



<a name="165250897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165250897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165250897">(May 09 2019 at 13:14)</a>:</h4>
<p>Since we are at version zero anyway, I'll take the easy route and just defer this decision.</p>



<a name="165250946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165250946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165250946">(May 09 2019 at 13:15)</a>:</h4>
<p>If we make the measureme suite a rustup component then we might decide to not support old formats at all.</p>



<a name="165254026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165254026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165254026">(May 09 2019 at 13:52)</a>:</h4>
<p>I had a thought about this yesterday after the meeting. Could we just make the first event in the file a "metadata" event by convention?</p>



<a name="165254036"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165254036" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165254036">(May 09 2019 at 13:52)</a>:</h4>
<p>We could then stuff a JSON blob or something in the string data for that event</p>



<a name="165254060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165254060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165254060">(May 09 2019 at 13:53)</a>:</h4>
<p>The deserializer would know to look at the first event and to try deserializing it's data</p>



<a name="165254077"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165254077" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165254077">(May 09 2019 at 13:53)</a>:</h4>
<p>If that fails, then we know it's an old style file</p>



<a name="165254091"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165254091" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165254091">(May 09 2019 at 13:53)</a>:</h4>
<p>If it succeeds, then we can pull the version from the JSON blob</p>



<a name="165262690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165262690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165262690">(May 09 2019 at 15:24)</a>:</h4>
<p>Could we just make a header "event" that's always, say, the first 64 bits of all files and put (for now) just the version info in there?</p>



<a name="165321243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165321243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165321243">(May 10 2019 at 09:17)</a>:</h4>
<p>In <a href="https://github.com/rust-lang/measureme/issues/15" target="_blank" title="https://github.com/rust-lang/measureme/issues/15">https://github.com/rust-lang/measureme/issues/15</a> I suggested to have a string table entry with JSON metadata</p>



<a name="165321303"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165321303" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165321303">(May 10 2019 at 09:18)</a>:</h4>
<p>however, the version determines how events are encoded, so we can't put that information into an encoded event</p>



<a name="165321329"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165321329" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165321329">(May 10 2019 at 09:19)</a>:</h4>
<p>we'd have to decode the event in order to get the version and we have to know the version in order to know how to decode the event</p>



<a name="165321415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165321415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165321415">(May 10 2019 at 09:20)</a>:</h4>
<p>I think a simple scheme, as implemented in <a href="https://github.com/rust-lang/measureme/pull/41" target="_blank" title="https://github.com/rust-lang/measureme/pull/41">https://github.com/rust-lang/measureme/pull/41</a>, where we reserve the first 8 bytes of each file for a header with just the version in it is a good choice.</p>



<a name="165321430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/binary%20format%20versioning/near/165321430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/binary.20format.20versioning.html#165321430">(May 10 2019 at 09:21)</a>:</h4>
<p>we only have to keep the encoding of the header stable, all other decoding can then dispatch on the version number</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>